Camera Mode Control

ABSTRACT

Cameras may monitor its operation and automatically switch between operation modes thereby to best capture users&#39; experiences. Auxiliary sensor data collected by the one or more sensors and/or captured image data may be analyzed to determine when a camera should switch to a different operation mode. The auxiliary sensor data and/or the content of the captured images may include motion information indicating whether the camera or the captured scene has relatively high-motion, relatively low-motion, or no motion. Event(s) may be detected by analyzing the auxiliary sensor data and/or analyzing captured image data, and preferred operation mode(s) suitable for capturing the events can be determined. Each preferred operation mode may be associated with a confidence value. A camera may switch to a preferred operation mode for capturing an event if the event is determined with high confidence to take place at a time point.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority under 35 U.S.C. §119(e) to U.S.Provisional Patent Application Ser. No. 62/155,882, “Camera ModeControl,” filed May 1, 2015, which is incorporated herein by referencein its entirety.

BACKGROUND

1. Technical Field

This disclosure relates to a camera, and more specifically, toautomatically controlling an operation mode of a camera.

2. Description of the Related Art

Most capture devices allow for a variety of options and configurationsfor capturing image, video, and audio content. For example, a capturedevice can be configured to capture a single image, a time-lapsed burstof multiple images, or video captured at varying frames per second(e.g., 24 frames per second (fps), 30 fps, 120 fps, 240 fps, etc.) In aconventional capture device, a user manually selects a specificoperation mode for capturing content. For action cameras used duringactivities such as skiing, snowboarding, surfing, biking, etc., usersoften do not have direct access to the camera or are otherwise focusedon the activities during the image or video capture. Users are often notin a position to switch camera operation modes at the right time tooptimize capturing the experience. In one representative examplescenario, a skier has cameras mounted on his helmet and his skis. Theskier sets up both cameras when starting his ski run, and at some pointduring his run, makes a jump. The user would like to capture this jumpwith the ski-mounted camera taking a burst of multiple images and thehelmet-mounted camera taking a high frame rate capture (for slow motioneffect), but the user is unable to manually switch either of the camerasto the desired modes in the middle of the ski run. Thus, conventionalcamera systems fail to provide any mechanism for automaticallydetermining and switching to an operation mode that best captures a userexperience.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present disclosure can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings.

FIG. 1 is a block diagram illustrating a camera, according to oneembodiment.

FIG. 2 is a block diagram illustrating a camera mode controller,according to one embodiment.

FIG. 3 is a flow chart illustrating a camera switching between operationmodes based on auxiliary sensor data, according to one embodiment.

FIG. 4 is a flow chart illustrating an example process for switching acamera between operating modes including an initial operation mode, apriming operation mode, and an alternative operation mode, according toone embodiment.

FIG. 5 is a timing diagram illustrating a camera switching betweenoperation modes according to confidence values associated with analternative operation mode (e.g., a high frame rate mode), according toone embodiment.

FIG. 6 is a flowchart illustrating an embodiment of a process forautomatically controlling a video enhancement feature based on motiondata.

FIG. 7 is a flowchart illustrating an embodiment of a process forautomatically controlling a capture mode of a plurality of differentcameras.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

Configuration Overview

Cameras may monitor its operation and automatically switch betweenoperation modes thereby to best capture users' experiences. Auxiliarysensor data collected by the one or more sensors and/or captured imagedata may be analyzed to determine when a camera should switch to adifferent operation mode. The auxiliary sensor data and/or the contentof the captured images may include motion information indicating whetherthe camera or the captured scene has relatively high-motion, relativelylow-motion, or no motion. Different events may require the camera tooperate at different operation modes to ensure image quality andminimize artifacts to best capture users' experiences. Event(s) may bedetected by analyzing the auxiliary sensor data and/or analyzingcaptured image data, and preferred operation mode(s) suitable forcapturing the events can be determined. Each preferred operation modemay be associated with a confidence value. A camera may switch to apreferred operation mode for capturing an event if the event isdetermined with high confidence to take place at a time point.

In one embodiment, a method controls operation of a camera includesreceiving, from one or more sensors of the camera, sensor data. Aconfidence value associated with an alternative operation mode foroperating the camera is determined based at least in part on thereceived sensor data. The confidence value indicates a likelihood of thealternative operation mode being preferred by a user to capture anevent. While the camera is operating in an initial operation mode inwhich an image sensor captures images according to an initial capturemode and an image processor reads the captured images from a buffer ofthe camera and encodes the captured images according to an initialencoding mode: responsive to determining that the confidence valueexceeds a first threshold, the camera is transitioned to a primingoperation mode. In the priming operation mode, the image sensor capturesimages according to an alternative capture mode and the image processorreads the captured images from the buffer of the camera and encodes thecaptured images according to the initial encoding mode. While the cameraoperating in the priming operation mode, responsive to determining thatthe confidence value subsequently drops below the first thresholdtransitioning the camera to the initial operation mode.

In one embodiment, a camera includes one or more sensors to generatesensor data. The camera further includes an image sensor to captureimages according to different capture modes including an initialoperation mode and an alternative operation mode, a buffer to store thecaptured images, and an image processor to read the captured images fromthe buffer and to encode the captured images according to differentencoding modes including an initial encoding mode and an alternativeencoding mode. The camera further includes a mode controller to receivethe sensor data from the one or more sensors and to determine aconfidence value associated with an alternative operation mode foroperating the camera based at least in part on the received sensor data,the confidence value indicating a likelihood of the alternativeoperation mode being preferred by a user to capture an event. While thecamera is operating in an initial operation mode in which the imagesensor captures images according to the initial capture mode and theimage processor reads the captured images from the buffer of the cameraand encodes the captured images according to the initial encoding mode,the mode controller transitions the camera to a priming operation moderesponsive to determining that the confidence value exceeds a firstthreshold. In the priming operation mode, the image sensor capturesimages according to an alternative capture mode and the image processorreads the captured images from the buffer of the camera and encodes thecaptured images according to the initial encoding mode. While the cameraoperating in the priming operation mode, the mode controller transitionsthe camera to the initial operation mode responsive to determining thatthe confidence value subsequently drops below the first threshold.

In one embodiment, a system includes a processor and a non-transitorymemory coupled to the processor. The memory store instructionsconfigured to cause the processor to receive, from one or more sensorsof the camera, sensor data. A confidence value associated with analternative operation mode for operating the camera is determined basedat least in part on the received sensor data, the confidence valueindicating a likelihood of the alternative operation mode beingpreferred by a user to capture an event. While the camera is operatingin an initial operation mode in which an image sensor captures imagesaccording to an initial capture mode and an image processor reads thecaptured images from a buffer of the camera and encodes the capturedimages according to an initial encoding mode: responsive to determiningthat the confidence value exceeds a first threshold, the camera istransitioned to a priming operation mode. In the priming operation mode,the image sensor captures images according to an alternative capturemode and the image processor reads the captured images from the bufferof the camera and encodes the captured images according to the initialencoding mode. While the camera operating in the priming operation mode,responsive to determining that the confidence value subsequently dropsbelow the first threshold transitioning the camera to the initialoperation mode.

Example Camera Configuration

FIG. 1 is a block diagram illustrating a camera 100, according to oneembodiment. In the illustrated embodiment, the camera 100 comprises acamera core 110 comprising a lens 112, an image sensor 114, and an imageprocessor 116. The camera 100 additional includes a system controller120 (e.g., a microcontroller or microprocessor) that controls theoperation and functionality of the camera 130 and system memory 130configured to store executable computer instructions that, when executedby the system controller 120 and/or the image processor 116, perform thecamera functionalities described herein. In some embodiments, a camera130 may include multiple camera cores 110 to capture fields of view indifferent directions which may then be stitched together to form acohesive image. For example, in an embodiment of a spherical camerasystem, the camera 130 may include two camera cores 110 each having ahemispherical or hyperhemispherical lens that each captures ahemispherical or hyperhemispherical field of view which are stitchedtogether in post-processing to form a spherical image.

The lens 112 can be, for example, a wide angle lens, hemispherical, orhyperhemispherical lens that focuses light entering the lens to theimage sensor 114 which captures images and/or video frames. In differentconfigurable operation modes, the image sensor 114 may capturehigh-definition images having a resolution of, for example, 3 Mp(megapixels), 5 Mp, 7 Mp, 8 Mp, 12 Mp, or higher or high-definitionvideo having resolutions of 720 p, 1080 p, 4 k, or higher. In a burstimage capture mode, the image sensor 114 may capture a burst of multipleimages at frame rates of, for example, 3 fps (frames per second), 5 fps,8 fps, 10 fps or higher. In different configurable operation modes, theimage sensor 114 may capture video frames at frame rates of, forexample, 24 fps, 30 fps, 60 fps, 120 fps, 240 fps, or higher. The imageprocessor 116 encodes the captured image data, for example, byperforming one or more image processing functions of the captured imagedata. For example, the image processor 116 may perform a Bayertransformation, demosaicing, noise reduction, image sharpening, imagestabilization, rolling shutter artifact reduction, color spaceconversion, compression, or other in-camera processing functions.Processed image data may be temporarily or persistently stored to systemmemory 130 and/or to a non-volatile storage, which may be in the form ofinternal storage or an external memory card. In one embodiment, imagedata captured by the image sensor 114 is temporarily stored to a buffer115. The image processor 116 reads the image data from the buffer 115and encodes the captured image data. Thus, the encoding lags behindcapture by a time period dependent on the processing times and thelength of the buffer 115.

Sensors 140 capture various sensor data concurrently with, orindependently of, image or video capture. The sensor data may be storedin association with the captured image or video data as metadata. Forexample, the sensors 140 may capture time-stamped location informationbased on a global positioning system (GPS) sensor, and/or an altimeter.Other sensors 140 may be used to detect and capture orientation of thecamera 100 including, for example, an orientation sensor, anaccelerometer, a gyroscope, or a magnetometer. Other sensors 140 may beused to detect and capture biometric information relating to a camerauser or subject of the capture such as, for example, pulse data. Sensordata captured from the various sensors 140 may be processed to generateother types of metadata. For example, sensor data from the accelerometermay be used to generate motion metadata, comprising velocity and/oracceleration vectors representative of motion of the camera 100.Furthermore, sensor data from the orientation sensor may be used togenerate orientation metadata describing the orientation of the camera100. Sensor data from the GPS sensor provides GPS coordinatesidentifying the location of the camera 100, and the altimeter measuresthe altitude of the camera 100. In one embodiment, the sensors 140 arerigidly coupled to the camera 100 such that any motion, orientation, orchange in location experienced by the camera 100 is also experienced bythe sensors 140. Alternatively, sensors 140 may be remote from thecamera 100 and affixed to an object of interest that is a subject of theimage or video capture (e.g., a bike, surfboard, vehicle, etc.). In anembodiment, the sensors 140 may be integrated with a mobile devicecarried by the user that can communicate wirelessly with the camera 100to provide the sensor data. In yet another embodiment, sensors may beintegrated with an aerial vehicle on which the camera 100 is mounted.The sensors 140 furthermore may associate a time stamp representing whenthe data was captured by each sensor. In one embodiment, the sensors 140automatically begin collecting sensor data when the camera 100 beginscapturing an image or recording a video.

An audio subsystem 150 includes, for example, one or more microphonesand one or more audio processors to capture and process audio datacorrelated with video capture. In one embodiment, the audio subsystem150 includes a microphone array having two or microphones arranged toobtain directional audio signals.

An input/output (I/O) interface 160 transmits and receives data fromvarious external devices. For example, the I/O interface 160 mayfacilitate the receiving or transmitting video or audio informationthrough an I/O port. Examples of I/O ports or interfaces include USBports, HDMI ports, Ethernet ports, audioports, and the like.Furthermore, embodiments of the I/O interface 160 may include wirelessports that can accommodate wireless connections. Examples of wirelessports include Bluetooth, Wireless USB, Near Field Communication (NFC),and the like. The I/O interface 160 may also include an interface tosynchronize the camera 100 with other cameras or with other externaldevices, such as a remote control, a second camera, a smartphone, aclient device, or a video server.

A control/display subsystem 170 includes various control and displaycomponents associated with operation of the camera 100 including, forexample, LED lights, a display, buttons, microphones, speakers, and thelike.

A mode controller 180 may control the camera 100 (and optionally one ormore additional cameras) to automatically switch between operation modesfor best capturing a user's experience. When operating in differentoperation modes, the camera 100 may generate images and/or videos havingdifferent properties. For example, the camera may capture images atdifferent resolutions (e.g., 18 Mp(megapixel), 14 Mp, 12 Mp, 8 Mp, 7 Mp,5 Mp, 3 Mp, etc.), or capture a series of images captured at differentrates (e.g., 3 fps (frames per second), 5 fps, 8 fps, 10 fps, 20 fps,etc.). Furthermore, video may be captured using different formats, framerates (e.g., 240 fps (frames per second), 120 fps, 60 fps, 30 fps, etc.)and/or resolutions (e.g., 1080 p, 730 p, 4K, etc.) Other camera functionsuch as aperture, shutter speed, exposure level, color correction,rolling shutter correction, etc. may also be automatically controlled.Different events may make it desirable for the camera to operate atdifferent operation modes to ensure image quality, prevent artifacts, orintroduce desired cinematic effects into the video. Events may bedetected by analyzing the auxiliary sensor data, content of the capturedimages and audio, control inputs (e.g., button presses, voice commands,gesture commands, etc.) detected by the camera, predefined userpreferences, camera state information (e.g., a current configuration ofthe camera), physical capabilities of the camera, or a combination ofother data collected by the mode controller 180. Then, one or moreoperation modes best suitable for capturing the event can be predicted.For example, the auxiliary sensor data and/or the content of thecaptured images and audio may include motion information indicatingwhether the camera or the captured scene has relatively high-motion,relatively low-motion, or no motion. The motion history may be used topredict future motion that is likely to take place and change theoperation mode accordingly. For example, if the mode controller 180detects a trend of increasing velocity, it may predict that the motionwill continue to increase and switch to a higher frame rate to enable aslow motion effect during playback. In another example, using locationinformation, the mode controller 180 may predict when the camera isentering or has entered a geographic area of interest. In yet anotherexample, the mode controller 180 analyzes biometric data from a user andchanges a camera mode in response to detecting an increased pulse rate.In one embodiment, the mode controller 180 continuously analyzes thevarious types of input data and generates a confidence score indicatinga likelihood of a particular event occurring for which it would bedesirable to trigger a mode change. In yet another example, the modecontroller 180 may switch operation modes in response to a button pressor control signal, either by itself or in combination with other inputs.

In one embodiment, the operation mode may be determined based at leastin part on the quantity of image sensors 114 and/or image processors 116available in a camera 100. For example, in a two camera arrangement, themode controller 180 may determine that a 4 k video mode is the preferredmode of operation, but only one of the two cameras has that capability.Thus, a different operation mode (e.g., 1080 p) may be selected for thecamera lacking the 4 k capability.

The mode controller 180 may switch the camera 100 to a determinedoperation mode for capturing an event if the event is determined to takeplace with high confidence (e.g., above a threshold confidence score).The mode controller 180 may be integrated with the camera 100 asillustrated or external to the camera 100. For example, the modecontroller 180 may be integrated with an external mobile device orremote control that wireless communicates with the camera. In anotherembodiment, the mode controller 180 may be integrated with a mountingplatform such as a handheld grip, a rotating mount, or an aerial vehiclethat communicates with and controls operation of the camera 100. In oneembodiment, the mode controller 180 (integrated with the camera 100 orexternal to the camera 100) may control operations of at least two ormore cameras. Here, mode switching of multiple cameras may becoordinated in order to produce desired cinematic effects when thecontent is combined in post-processing.

The different possible operation modes of one or more cameras that themode controller 180 selects between and the criteria for switchingbetween the modes may be pre-configured or selectable by the user.Furthermore, if the mode controller 180 controls multiple cameras 100,the mode for each camera 100 at a given time may be different and thecriteria for switching between camera modes in each camera 100 may bedifferent. For example, a user may configure various predefined modeshaving a combination of settings (e.g., resolution, frame rate, burstmode (e.g., the number of images captured within a second), or aparticular camera, etc.) for capturing events of different types.

FIG. 2 is a block diagram illustrating a camera mode controller 180,according to one embodiment. The camera mode controller 180 comprises asensor and content analytics module 202, and a command and controlmodule 204. The sensor and content analytics module 202 analyzesauxiliary data, image content, audio content, control inputs, or otherinput data and determines preferred operation modes for operating one ormore cameras. The command and control module 204 controls operations ofthe one or more cameras 100 based on the operation mode determined bythe sensor and content analytics module 202. For example, the commandand control module 204 regulates operations of the image sensor 114,image processor 116, or other components of the camera 100 according tothe operation modes determined by the sensor analytics module 202. Moredetails of determining operation modes and regulating a camera'soperation according to the determined operation modes are provided belowwith respect to FIGS. 3-5.

Camera Mode Control

FIG. 3 is a flow chart illustrating an embodiment of a process forswitching a camera between operation modes, according to one embodiment.A mode controller 180 controls a camera 100 to operate 302 in an initialoperation mode in which its image sensor(s) capture images according tothe initial operation mode and its image processor(s) encode images (orvideo frames) according to the initial operation mode.

The camera 100 receives 304 input data, which may include sensor datafrom one or more sensors 140 (e.g., motion information, locationinformation, biometric information, temperature, proximity to othercameras, etc.), the audio/visual data captured by the camera 100, one ormore control inputs (e.g., a button press, voice command, gesturecommand, etc.), user preferences, camera state information (e.g., acurrent operation mode of the camera 100), physical capabilities of thecamera, or a combination of these or other factors. The mode controller180 (e.g., the sensor analytics module 202) determines one or moreoperation modes for operating the camera 100 based at least in part onthe received data. For example, in one embodiment, the mode controller180 determines scores for a plurality of different possible operatingmodes based on the received data and generates a ranked list ofoperating modes. The ordered operation mode(s) may represent a rankingof the operation modes most likely to be desired by the user to operatethe camera to capture the events determined based on the received inputdata. The sensor and content analytics module 202 may analyze thereceived input data to detect events or likelihood of events that aretaking place. In one embodiment, the sensor and content analytics module202 may determine a confidence value associated with an eventrepresenting a likelihood of the event taking place. For example, thesensor and content analytics module 202 determines a velocity of thecamera 100 by dividing the distance between the locations of the lastand current time stamps over the time duration between the last andcurrent time stamps. An acceleration of the camera 100 may be determinedby dividing the velocities of the last and current time stamps over thetime duration between the last and current time stamps. The sensor andcontent analytics module 202 determines that an increase in accelerationwhen the acceleration is above a threshold acceleration indicates ahigh-motion event and assigns a high confidence value to the high-motionevent. In addition, the sensor and content analytics module 202 mayfurther assign a low confidence value to a mid-motion event or alow-motion event. As one example, an increase in acceleration above athreshold acceleration may indicate a skier approaching a jump, which isa high-motion event. In another example, the confidence value mayincrease with increasing pulse rate. In yet another example, theconfidence value may increase with increasing altitude or as the userapproaches a location of interest. In yet another embodiment, theconfidence value may be set to a first value when a voice or gesturecommand is first detected, and set to a second higher value when thevoice or gesture command is confirmed.

In one embodiment, the mode controller 180 receives a n-dimension sensorinput, and classifies the received sensor input into K possible cameraoperation mode clusters. Correlations between K possible cameraoperation modes and a range of n dimensional sensor vectors areevaluated by a classifier. The classifier is trained by using a trainingsample set of input videos and the corresponding operation modes. Givena particular N dimensional input sensor array {S(0) . . . S(n)}, itsdistance from each of the K possible operation modes is calculated. Thedistances can be represented by a vector {d(0) . . . d(k)} andnormalized such that Dn(i)=d(i)/Sum(d(0) . . . d(k)).

The inverse of Dn(i) measures how strongly the sensor vector correlateswith a particular camera mode K. The operation modes are furtherfiltered and rescored based on the camera's state such as the camera'sbattery health, temperature, and other parameters. The operation modewith the strongest score is selected for determining a first thresholdvalue TH₁ and a second threshold value TH₂. The threshold values TH₁ andTH₂ measure how far apart the strongest score is with respect to themedian.

In further embodiments, the confidence value can be based on other inputdata or a combination of different types of input data. In amulti-camera arrangement, different operation modes may be determinedfor different cameras based on the received input data.

One or more cameras' operation is controlled 308 according to thedetermined operation mode(s). Here, the command and control module 204may generate 310 an output to control operations of the camera 100 orgenerate a control signal to cause one or more external devices (e.g.,another camera, camera accessory, or mount platform) according to thedetermined operation mode. The mode controller 180 receives sensor inputfrom all cameras involved in the multicamera control and calculatesoperation modes for all the cameras. For each operation mode X(i), acamera C(i) is assigned such that the sensor vector V(i) thatcorresponds to the camera is most strongly correlated with the operationmode X(i), compared to the other cameras.

Control of Capture Frame Rate

FIG. 4 is a flow chart illustrating an example process for switching acamera between operating modes including an initial operation mode, apriming operation mode, and an alternative operation mode, according toone embodiment. In this embodiment, the initial operating moderepresents, for example, a standard frame rate capture mode (e.g., 30frames per second) in which the image sensor 114 captures images at thestandard frame rate, stores the frames to a buffer 115, and the imageprocessor 116 reads the frames from the buffer 115 and encodes theframes at the standard frame rate. The alternative operating moderepresents, for example, a high frame rate capture mode (e.g., 60 fps).The priming mode represents a mode in which the image sensor 114operates at the high frame rate (e.g., 60 fps) but the image processor116 encodes the images at the standard frame rate (e.g., 30 fps). Thepriming mode is generally used as a transitional mode between theinitial mode and the alternative mode, the advantages of which will bedescribed in further detail below.

Operations of the image sensor 114 and the image processor 116 arecontrolled in this example embodiment by generating confidence valuesassociated with an event for which a high frame rate capture isdesirable. For example, a higher confidence value may be generated asthe acceleration or velocity increases, as altitude increases, as pulserate increases, as a location of interest is approached, as inputcommands are received, or based on a combination of these or otherfactors.

In some embodiments, operation of the image processor 116 may becontrolled to track the operation of the image sensor 114. The operationof the image sensor is controlled based on the determined confidencevalue associated with an event for which a high frame rate capture isdesirable. The image processor 116 is controlled to delay switching itsoperation by a predetermined duration (e.g., n seconds of buffer size)subsequent to the image sensor 114 switching its operation. Thepredetermined duration is a time period the image processor 116 needs toencode buffered data. This mechanism accounts for the delay inevaluating that such a moment has occurred since the measurements andanalysis are performed on events that have already occurred. Thisenables the camera to apply the new encoding configuration retroactivelyon buffered data.

The high frame rate event may possibly not take place after the cameraswitching to the priming mode. To save the battery life, the camera 100may transition back to the initial operation mode at a predeterminedtime duration (e.g., n seconds of buffer size) subsequent to switchingthe image sensor 114 to operate at the alternative operation mode,responsive to determining that the confidence value is still greaterthan a first value but less than a second higher value. Furthermore, inone embodiment, a user can provide explicit input by means of voice orgesture or a button press, etc., indicating that an event is about tooccur that would warrant switching to a different operation mode.

A camera operates 402 in the standard frame rate mode in which thecamera captures and encodes image data according to baseline settingsthat may be user-defined or hard coded. While operating in the baselineframe rate mode, the camera determines confidence values associated withtransitioning to the high frame rate mode based at least in part onsensor data received from sensors, the image or video content, controlsignals, or other input data. The high frame rate mode may be auser-defined or hard coded frame rate higher than the standard framerate.

While operating in the standard frame rate mode, the camera compares 404the determined confidence value to a first threshold value TH₁.Responsive to determining that the confidence value is below the firstthreshold value TH₁, the camera continues to operate in the standardframe rate mode. Responsive to determining that the confidence valueexceeds the first threshold TH₁, the camera switches to operate 406 inthe priming mode. While operating in the priming mode, the imageprocessor 116 encodes, according to the standard frame rate mode (e.g.,encoding at 30 fps), the buffered data that is captured according to thehigh frame rate mode (e.g., captured at 60 fps or 120 fps). Whileoperating in the priming operation mode, the camera compares 408 adetermined confidence value to the first threshold value TH₁. The cameratransitions back to the standard frame rate mode responsive todetermining that the determined confidence value falls below the firstthreshold value TH₁.

The camera also compares 410 the determined confidence value to a secondthreshold value TH₂ and continues to operate in the priming operationmode responsive to determining that the confidence value does not exceedthe second threshold value TH₂. The camera transitions to operating 412in the high frame rate mode responsive to determining that theconfidence value exceeds the second threshold. Here, the image sensor114 continues to operate according to the high frame rate mode and theimage processor 116 switches to operate according to the high frame ratemode. Because the encoding lags behind the image capture (due to thebuffering of the captured frames), the image processor may either beginencoding at the high frame rate after it completes encoding the alreadybuffered frames at the low frame rate mode, or the image processor mayencode all or some of the buffered frames at the high frame rate. Whileoperating in the high frame rate mode, the camera continues to generateconfidence values and compares 414 the confidence values to the firstthreshold value TH₁. The camera transitions back to the standard framerate mode responsive to determining the confidence value drops below thefirst threshold value TH₁. The image sensor switches to operateaccording to the low frame rate mode responsive to determining that theconfidence value drops below the first threshold. The image processormay switch to operate according to the standard frame rate mode aftercompleting encoding of the already buffered frames at the high framerate mode, or the image processor may encode all or some of the bufferedframes at the standard frame rate.

The use of buffering in combination with the priming mode fortransitioning between the standard frame rate mode and the high framerate mode beneficially enables the camera to encode frames at the highframe rate even for frames that were captured before the secondthreshold is crossed, indicating a high confidence that an event isoccurring that would be desirable to capture at the high frame rate.Because there may be some delay between when the relevant event actuallyoccurs and when the processed input data reflects a confidence valueexceeding the second threshold, the camera can still ensure that theevent is not missed. Furthermore, the camera can save processing powerand memory resources by only encoding at the high frame rate when thereis high confidence that the event of interest is occurring.

FIG. 5 is a timing diagram illustrating an example in which a cameraswitches between a standard frame rate mode, a priming mode, and a highframe rate mode according to confidence values associated with alikelihood of an event occurring for which it is desirable to operate ahigh frame rate. At time t₀, the camera 100 operates in the initialoperation mode (e.g., a standard frame rate mode). The image sensor 114starts capturing image data according to the standard frame rate mode.The captured image frames are buffered, for example, in an FIFO buffer.At time t₁, the buffer 115 becomes full and the image processor 116begins encoding the image frames from the buffer 115, also at thestandard frame rate. As can be seen, the image processor 116 lags behindthe image sensor 114. Thus, for example, a frame captured at time t₀ isencoded at time t₁. The time duration (t₁-t₀) is the time differencebetween the image sensor 114 beginning to capture and store image datain a buffer 115 and the image processor 116 beginning to encode thebuffered image data from the buffer 115.

At time t₂, the confidence value associated with the high frame ratemode begins to increase. For example, a velocity, acceleration, pulserate, or altitude of a skier begins to increase indicating the skier isnearing a jump. Nevertheless, because the confidence value has not yetreached the first threshold TH₁, the camera 100 remains operating in theinitial operation mode.

At time t₃, the confidence value associated with the high frame ratemode reaches the first threshold TH₁. The camera 100 switches itsoperation from the initial operation mode to a priming operation mode.In the priming operation mode, the image sensor 114 switches to captureimage data according to the high frame rate and stores the high framerate data to the buffer 115. The image processor 116 continues to encodethe buffered data according to the standard frame rate mode.

At time t₄, the confidence value associated with the event of interestreaches a second threshold TH₂ indicating with very high confidence thatthe event of interest is occurring and that it is desirable to encodethis content at the high frame rate. At time t₅, the frames captured bythe image sensor at t₄ are read from the buffer and encoded. Between t₄and t₅, the image processor 116 may optionally continue to encode at thestandard frame rate (so that the frames captured when the confidencevalue is between TH₁ and TH₂ are encoded at the standard frame rate), orthe image processor 116 may encode all or some of the buffered data atthe high frame rate (so that all or some of the frames captured when theconfidence value is between TH₁ and TH₂ are encoded at the high framerate). Additionally, if the time between t₃ and t₄ is longer than thebuffer time, there may be some frames capturing during the priming modethat were already encoded at the standard frame rate. In this case,these frames may optionally be re-encoded (e.g., at time t₄) at the highframe rate if the buffer permits. In this way, frames before the highconfidence threshold TH₂ is reached can be encoded at a high frame rate,thus ensuring that the event of interest is not missed.

At the time t₆, the confidence value drops below the second thresholdTH₂. The image sensor 114 and the image processor 116 both continue tooperate according to the high frame rate mode. The confidence valueassociated with the high frame rate mode continues to drop, and at atime t₇, the confidence value drops below the first threshold TH₁indicating with reasonably high certainty that the event of interest isover. The image sensor 114 returns to operate according to the standardframe rate mode. The image processor 116 may continue to encode thealready buffered data captured by the image sensor 114 while in the highframe rate mode until it completes encoding the buffered data at timet₈, or the image processor 116 may switch to encode at the standardframe rate at time t₇. Additionally, all or some of the frames capturedbetween t₆ and t₇ could be optionally re-encoded at the standard framerate after the confidence value drops below TH₁ and it is determined theframes captured during this time were likely after the event ofinterested ended, After time t₈, the camera returns to the initialoperation mode.

Controlling Video Enhancement Feature

In one embodiment, the confidence value in FIG. 6 may track velocity,acceleration, altitude, pulse rate, or a combination of parameters. Inanother embodiment, the confidence value may be set to a value betweenTH₁ and TH₂ in response to a first command being received via a voice,gesture, or button input, set to a value above TH₂ in response toconfirming the command (e.g., by issuing a prompt for the user toconfirm the command and the user providing an input to respondaffirmatively), and set to a value below TH₁ in response to a secondcommand. This embodiment enables a user to manually control the cameramode.

In another embodiment, the mode controller 180 intelligently turns on oroff a video enhancement feature. For example, some features such asimage stabilization or rolling shutter artifact reduction are mostbeneficial in videos with relatively high motion. If the motion isrelatively low, these enhancements will provide little improvement tothe video quality. Furthermore, because applying image stabilization,rolling shutter artifact reduction, or other enhancement algorithms cansignificantly increase power consumption, it may be desirable to enablethese features only when they will be provide a significant improvementin quality (e.g., when motion is detected to be above a threshold level)and are otherwise disabled to reduce power.

In one example of a rolling shutter artifact reduction algorithm, videoframes are captured by the image sensor 114 at a higher frame rate thanthe image processor 116 reads them out. For example, the image sensor114 may capture at 120 frames per second but the image processor 116reads at 30 frames per second, thus reading every 4 frames or combiningevery 4 frames into a single frame. Alternatively, the image sensor 114may capture an image during a relatively short capture interval (e.g.,equivalent to a 120 fps capture interval) and then waits for a blankinginterval (e.g., the length of 3 120 fps capture intervals) beforecapturing the next frame. In either case, the capture time is relativelyshort compared to the frame rate. The shortened capture time reduces therolling shutter artifact. An embodiment of an algorithm for reducingrolling shutter artifact is described in, for example, U.S. Pat. No.8,830,367 which is incorporated by reference herein.

In other enhancement algorithms such as video stabilization, the imagesensor 114 overcaptures a video window so that the capture frame islarger than the output view window. Motion can be then be compensated byshifting the position of the output view window opposite to thedirection of detected motion.

FIG. 6 illustrates an embodiment of a process for controlling a videoenhancement feature (e.g., rolling shutter artifact reduction or imagestabilization) based on detected motion. Sensor data or motion analysisfrom the image content itself is received 602 and a motion level isdetected 604. The motion level may comprise, for example, a detectedvelocity or acceleration of the camera relative to the scene. The motionlevel is compared 606 to a motion threshold. If the motion level is lessthan the threshold, the video enhancement feature is disabled 608 toreduce power consumption. If the motion level is greater than thethreshold, the video enhancement feature is enabled 610 to reduce motionartifacts. The process may repeat to enable or disable the videoenhancement feature on a frame-by-frame basis or for a range of frames.

Automatically Controlling Operation Mode of Multiple Cameras

As described earlier, in one embodiment a mode controller 180 cancontrol the operation mode of a plurality of different camerascommunicatively coupled to the mode controller 180. FIG. 7 illustrateson embodiment of a process for controlling operating modes of multiplecameras. The mode controller 180 receives 702 the capabilities of eachof the connected cameras. For example, the mode controller 180 maydetermine what resolutions and frame rates each camera is capable ofoperating at. Furthermore, the mode controller 180 may determine howmany image processors 116 and image sensors 114 are available at eachcamera. The mode controller 180 analyzes 704 input data (e.g., sensordata, image/audio data, control input data, etc.) and determines 706 ascore for each of a plurality of predefined operating modes. The scoresmay be computed, for example, based equations or a set of rules that mapthe set of inputs to a score for each predefined mode indicating howdesirable that operation mode is given the input parameters. The modecontroller 180 generates a ranked list of operating modes ordered basedon their scores. For example, the mode controller 180 may determine thata 1080 p resolution 120 fps video is a first preferred operation mode, a1/30 burst picture is a second preferred operation mode, and a 4 kpresolution 30 fps video is a third preferred operation mode. The modecontroller 180 then assigns the camera modes to each of the connectedcameras based on their capabilities. For example, starting with thefirst preferred operating mode, the mode controller 180 determines if acamera is available that is capable of operating in that mode, and ifso, assigns the camera to that mode. The mode controller 180 thensimilarly assigns the second preferred operating mode to another cameraif available and capable of operating in the mode. In some embodiments,and integrated camera system (e.g., a spherical camera) may include adifferent number of image sensors 114 and image processors 116 (e.g.,two image sensors 114 linked to a single image processor 116). In thesecases, the mode controller 180 may assign operating modes to the imagesensors 114 and image processors 116 separately within a single camera.For example, one image processor may be configured to encode at a highframe rate while the other is configured to encode a low frame rate.

The technique of FIG. 7 beneficially enables multiple cameras to beconfigured in different modes to capture an event according to more thanone set of capture parameters instead of each camera independentlycapturing the event with the same parameters. For example, when goingover a ski jump, it may be desirable to capture the jump with both highframe rate video and a burst image capture. This enables the user todetermine in post-processing which capture is most desirable or lets theuser combine footage from a combination of different captures fromdifferent cameras.

Additional Configuration Considerations

Throughout this specification, some embodiments have used the expression“coupled” along with its derivatives. The term “coupled” as used hereinis not necessarily limited to two or more elements being in directphysical or electrical contact. Rather, the term “coupled” may alsoencompass two or more elements are not in direct contact with eachother, but yet still co-operate or interact with each other, or arestructured to provide a thermal conduction path between the elements.

Likewise, as used herein, the terms “comprises,” “comprising,”“includes,” “including,” “has,” “having” or any other variation thereof,are intended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus.

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the invention. Thisdescription should be read to include one or at least one and thesingular also includes the plural unless it is obvious that it is meantotherwise.

Finally, as used herein any reference to “one embodiment” or “anembodiment” means that a particular element, feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. The appearances of the phrase “in oneembodiment” in various places in the specification are not necessarilyall referring to the same embodiment. Upon reading this disclosure,those of skill in the art will appreciate still additional alternativestructural and functional designs for the described embodiments asdisclosed from the principles herein. Thus, while particular embodimentsand applications have been illustrated and described, it is to beunderstood that the disclosed embodiments are not limited to the preciseconstruction and components disclosed herein. Various modifications,changes and variations, which will be apparent to those skilled in theart, may be made in the arrangement, operation and details of the methodand apparatus disclosed herein without departing from the scope definedin the appended claims.

What is claimed is:
 1. A method for controlling operation of a camera, comprising: receiving, from one or more sensors of the camera, sensor data; determining a confidence value associated with an alternative operation mode for operating the camera based at least in part on the received sensor data, the confidence value indicating a likelihood of the alternative operation mode being preferred by a user to capture an event; while the camera is operating in an initial operation mode in which an image sensor captures images according to an initial capture mode and an image processor reads the captured images from a buffer of the camera and encodes the captured images according to an initial encoding mode: responsive to determining that the confidence value exceeds a first threshold, transitioning the camera to a priming operation mode in which the image sensor captures images according to an alternative capture mode and the image processor reads the captured images from the buffer of the camera and encodes the captured images according to the initial encoding mode; and responsive to determining that the confidence value subsequently drops below the first threshold while the camera operating in the priming operation mode, transitioning the camera to the initial operation mode.
 2. The method of claim 1, further comprising: while the camera is operating in the priming operation mode, responsive to determining that the confidence value subsequently exceeds a second threshold greater than the first threshold, transitioning the camera to the alternative operation mode in which the image sensor captures images according to the alternative capture mode and the image processor reads the captured images from the buffer of the camera and encodes the captured images according to an alternative encoding mode.
 4. The method of claim 2, wherein transitioning the camera to the alternative operation mode comprises: encoding, by the image processor according to the initial encoding mode, images already stored in the buffer that were captured by the image sensor with the camera operating in the priming mode; and after the image processor encodes the captured images from the buffer, transitioning the image processor to operate in the alternative encoding mode.
 5. The method of claim 2, wherein transitioning the camera to the alternative operation mode comprises: encoding, by the image processor according to the alternative encoding mode, at least a subset of images already stored in the buffer that were captured by the image sensor with the camera operating in the priming mode; and after the image processor encodes the images from the buffer, operate the image processor in the alternative encoding mode.
 6. The method of claim 2, further comprising: while the camera is operating in the alternative operation mode, responsive to determining that the confidence value subsequently drops below the first threshold, transitioning the camera to the initial operation mode.
 7. The method of claim 6, wherein transitioning the camera to the initial operation mode comprises: encoding, by the image processor according to the alternative encoding mode, images already stored in the buffer that were captured by the image sensor with the camera operating in the alternative mode; and after the image processor encodes the images from the buffer, transitioning the image processor to operate in the initial encoding mode.
 8. The method of claim 6, wherein transitioning the camera to the initial operation mode comprises: encoding, by the image processor according to the initial encoding mode, at least a subset of images already stored in the buffer that were captured by the image sensor with the camera operating in the priming mode; and after the image processor encodes the images from the buffer, operate the image processor in the initial encoding mode.
 9. The method of claim 1, further comprising: determining confidence values for each of a set of predefined operation modes; ranking a set of predefined operation modes according to the associated confidence values; and determining the alternative operation mode having a highest confidence value.
 10. The method of claim 9, further comprising: identifying a first quantity of image sensors that are available and a second quantity of image processors that are available; and determining the set of predefined operation modes for the camera based at least in part on the first quantity and the second quantity.
 11. The method of claim 1, wherein the camera transitions to the priming operation mode at a first time point, further comprising: transitioning the camera to the alternative operation mode in which the image sensor and the image encoder operate according to the alternative operation mode at a second time point later than the first time point by a predetermined duration.
 12. The method of claim 1, wherein the camera transitions to the priming operation mode at a first time point, further comprising: responsive to determining the confidence value subsequently greater than the first threshold but below a second threshold greater than the first threshold, transitioning the camera to the initial operation mode in which the image sensor and the image processor operate according to the initial operation mode at a second time point later than the first time point by a predetermined duration.
 13. The method of claim 1, wherein the confidence value associated with an alternative operation mode for operating the camera is determined by providing the received sensor data to a classifier that is trained to evaluate correlations between the received sensor data and a set of alternative operation modes.
 14. A camera, comprising: one or more sensors to generate sensor data; an image sensor to capture images according to different capture modes including an initial operation mode and an alternative operation mode; a buffer to store the captured images; an image processor to read the captured images from the buffer and to encode the captured images according to different encoding modes including an initial encoding mode and an alternative encoding mode; and a mode controller to: receive the sensor data from the one or more sensors; determine a confidence value associated with an alternative operation mode for operating the camera based at least in part on the received sensor data, the confidence value indicating a likelihood of the alternative operation mode being preferred by a user to capture an event; while the camera operating in an initial operating mode in which the image sensor captures images according to the initial capture mode and the image processor reads the captured images from the buffer and encodes the captured images according to the initial encoding mode, responsive to determining that the confidence value exceeds a first threshold, transition the camera to a priming operation mode in which the image sensor captures images according to the alternative capture mode and the image processor reads the captured images from the buffer and encodes the captures images according to the initial encoding mode; and while the camera operating in the priming operation mode, responsive to determining that the confidence value subsequently drops below the first threshold, transition the camera to the initial operation mode.
 15. The camera of claim 14, wherein the mode controller is further configured to: while the camera is operating in the priming operation mode, responsive to determining that the confidence value subsequently exceeds a second threshold greater than the first threshold, transition the camera to the alternative operation mode in which the image sensor captures images according to the alternative capture mode and the image processor reads the captured images from the buffer of the camera and encodes the captured images according to an alternative encoding mode.
 16. The camera of claim 15, wherein transitioning the camera to the alternative operation mode comprises: encoding, by the image processor according to the initial encoding mode, images already stored in the buffer that were captured by the image sensor with the camera operating in the priming mode; and after the image processor encodes the captured images from the buffer, transitioning the image processor to operate in the alternative encoding mode.
 17. The camera of claim 15, wherein transitioning the camera to the alternative operation mode comprises: encoding, by the image processor according to the alternative encoding mode, at least a subset of images already stored in the buffer that were captured by the image sensor with the camera operating in the priming mode; and after the image processor encodes the images from the buffer, operate the image processor in the alternative encoding mode.
 18. The camera of claim 15, wherein the mode controller is further configured to: while the camera is operating in the alternative operation mode, responsive to determining that the confidence value subsequently drops below the first threshold, transitioning the camera to the initial operation mode.
 19. The camera of claim 18, wherein transitioning the camera to the initial operation mode comprises: encoding, by the image processor according to the alternative encoding mode, images already stored in the buffer that were captured by the image sensor with the camera operating in the alternative mode; and after the image processor encodes the images from the buffer, transitioning the image processor to operate in the initial encoding mode.
 20. The camera of claim 18, wherein transitioning the camera to the initial operation mode comprises: encoding, by the image processor according to the initial encoding mode, at least a subset of images already stored in the buffer that were captured by the image sensor with the camera operating in the priming mode; and after the image processor encodes the images from the buffer, operate the image processor in the initial encoding mode.
 21. The camera of claim 14, wherein the mode controller is further configured to: determine confidence values for each of a set of predefined operation modes; rank a set of predefined operation modes according to the associated confidence values; and determine the alternative operation mode having a highest confidence value.
 22. The camera of claim 21, wherein the mode controller is further configured to: identify a first quantity of image sensors that are available and a second quantity of image processors that are available; and determine the set of predefined operation modes for the camera based at least in part on the first quantity and the second quantity.
 23. The camera of claim 14, wherein the camera transitions to the priming operation mode at a first time point and the mode controller is further configured to transition the camera to the alternative operation mode in which the image sensor and the image processor operate according to the alternative operation mode at a second time point later than the first time point by a predetermined duration.
 24. The camera of claim 14, wherein the camera transitions to the priming operation mode at a first time point and the mode controller is further configured to: responsive to determining the confidence value subsequently greater than the first threshold but below a second threshold greater than the first threshold, transitioning the camera to the initial operation mode in which the image sensor and the image processor operate according to the initial operation mode at a second time point later than the first time point by a predetermined duration.
 25. The camera of claim 14, wherein the confidence value associated with an alternative operation mode for operating the camera is determined by providing the received sensor data to a classifier that is trained to evaluate correlations between the received sensor data and a set of alternative operation modes.
 26. A camera, comprising: one or more sensors; a processor; and a non-transitory memory coupled to the processor and storing instructions configured to cause the processor to: receive, from the one or more sensors of the camera, sensor data; determine a confidence value associated with an alternative operation mode for operating the camera based at least in part on the received sensor data, the confidence value indicating a likelihood of the alternative operation mode being preferred by a user to capture an event; while the camera is operating in an initial operation mode in which an image sensor captures images according to an initial capture mode and an image processor reads the captured images from a buffer of the camera and encodes the captured images according to an initial encoding mode: responsive to determining that the confidence value exceeds a first threshold, transitioning the camera to a priming operation mode in which the image sensor captures images according to an alternative capture mode and the image processor reads the captured images from the buffer of the camera and encodes the captured images according to the initial encoding mode; and responsive to determining that the confidence value subsequently drops below the first threshold while the camera operating in the priming operation mode, transitioning the camera to the initial operation mode. 